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INTRODUCTION 


General  Overview 

The  purpose  of  this  paper  is  to  document  the  efforts  of  a  major 
Department  of  Defense  (DoD)  development  and  acquisition  agency  to  develop  a 
workable,  valid  productivity  indicator  upon  which  to  base  critical  manpower 
decisions.  This  particular  agency  produces  highly  complex  weapon  systems 
that  require  years  of  development  and  acquisition  time  and  therefore 
require  a  relatively  sophisticated  objective  measure  upon  which  to  measure 
relative  requirements  across  differing  system  acquisition  offices. 

During  the  development  phase,  the  agency  selects  contractors  to 
produce  prototypes  for  test  and  evaluation  and  later  for  manufacturing  the 
quantities  the  DoD  orders  for  inventory.  During  development,  a  program 
manager  oversees  all  the  activities  of  contracting,  testing,  and  procuring 
the  munitions.  The  tasks  of  development  may  involve  coordinating  with  20 
or  more  organizations  within  and  outside  the  DoD.  Such  a  large  management 
effort  requires  a  large  amount  of  manpower  of  a  variety  of  skills.  The 
workload  is  uneven  over  a  period  of  time,  which  often  exceeds  five  years. 

The  total  manpower  depends  upon  the  stage  in  the  development  cycle, 

> 

unprogrammed  difficulties,  and  priority  of  the  program. 

The  measurement  of  productivity  is  not  a  simple  process,  and  is 
particularly  difficult  within  many  military  settings  wherein  the  ratio  of 
inputs  to  outputs  is  not  always  practical  (Cook,  1982).  Thorn  (1981,  p.  31) 
correctly  asserts  that  "outputs  frequently  cannot  be  measured  in 
quantitative  terms  and  that  multiple  objectives  of  a  disparate  nature 
preclude  the  use  of  a  single  measure."  In  this  case,  weapon  acquisition 
and  development  does  result  in  a  measurable  end  product.  However,  these 
products  are  usually  complex,  time  consuming,  and  dependent  upon  decisions 
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beyond  the  control  of  local  management.  It  is  therefore  important  to 
provide  some  intermediate  indicator  of  effectiveness. 

If  such  a  measure  were  available,  the  performance  of  manpower  overall 
(or  by  breakout  of  categories)  could  be  determined  and  changes  made  in 
aggregate  manpower  which  would  improve  the  overall  productivity  of  the 
organization.  The  following  describes  one  attempt  to  construct  and 
validate  a  productivity  measure  for  the  purpose  of  aggregate  manpower 
planning.  It  is  important  to  note  that  this  use  of  the  analytical  hierarchy 
process  to  construct  a  productivity  measure  is  a  unique  extension  of 
Saaty's  (1982)  method  for  evaluating  weights  of  multiattribute  models. 
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Literature 


In  complex  problems  such  as  manpower  planning  in  large  organizations, 
simple  procedures  as  decompositions  are  desirable  to  aid  the  decision 
maker.  Theoretical  multiattribute  utility  models  (MAU)  have  been 
developed  which  hold  promise  of  helping  the  decision  maker  in  practice. 
Keeney  and  Raiffa  (1976)  summarized  the  practical  procedures  for  developing 
MAU  models. 

The  simplest  utility  function  for  n  attributes  is  the  linear 
addi ti ve  model : 

u(x)  =  r  k.u.(x  ), 
i=l  1  1  1 

where  x  =  n-vector  of  attribute  values  x.,x0,..,x  , 

1  2  *  n’ 

u(x)  =  utility  of  the  vector  x, 
ki  =  weighting  of  attribute  X.,  i=l,l,...,n, 
x^  =  a  value  of  the  attribute  X., 
ui(xi)  =  marginal  utility  of  x„.. 

Fishburn  (1965)  proved  that  the  additive  utility  model  for  two 
attributes,  say  X  and  Y,  is  applicable  if  and  only  if  X  and  Y  are  additive 
independent.  Attributes  X  and  Y  are  additive  independent  if  the  paired 
preference  comparison  of  any  two  lotteries,  defined  by  two  joint 
probability  distributions  on  X  and  Y,  depends  only  on  their  marginal 
probability  distributions. 

Fishburn  extended  the  two  attribute  result  for  additive  indepen¬ 
dence  to  the  general  n  attribute  case.  The  utility  function  takes  the 
additive  form  if  and  only  if  the  set  of  attributes  (Xj,X2,...,X  )  is  addi¬ 
tive  independent,  that  is,  if  and  only  if  preference  over  all  lotteries  on 
(X1,X2>. . .  ,Xn)  depend  only  upon  their  marginal  probability  distributions. 
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The  multilinear  utility  function  is  applicable  only  when  the 
attributes  are  jointly  utility  independent.  A  set  of  attributes  is  defined 
to  be  jointly  utility  independent  if  and  only  if  each  of  the  attributes  is 
utility  independent  of  its  complement  (the  remaining  attributes). 

If  the  attributes  (X^  ,X2 , . . .  ,Xn)  are  mutually  utility  independent, 
then  the  utility  function  u(x)  takes  the  multiplicative  form. 

Mx)  +1  =  Z  (kk.u,(x.)  +  1), 
i=l  1  1  1 

u(x)  is  normalized  by  u(x1  ,x2 , . . .  ,xn)  =  0  and 
u(x^  ,X^  ,  . . .  ,Xn)  =  1, 

ui(xi )  is  a  marginal  (conditional)  utility 
function  on  X^  normalized  by  (x.)  =  0  and 
u1(x1)  =  1,  i  =  1,2,. ..,n, 
k-j  =  u(xi,x.), 

k  is  a  scaling  constant  determined  by  an  interactive 
procedure  given  in  Keeney  (1976). 

When  the  sum  of  the  n  scaling  constants  (the  k.'s)  is  equal  to 
one,  the  multiplicative  form  reduces  to  the  additive  model.  This 
simplification  is  proven  in  Keeney  (1976). 

Thomas  L.  Saaty  (1982)  has  pioneered  an  alternate  method  for 
evaluating  the  weights  of  multiattribute  models  using  direct  comparisons  of 
attributes.  This  process  is  called  the  analytical  hierarchy  process. 

Saaty  makes  the  comparisons  and  determines  the  weights  by  finding  the 
normalized  eigenvector  associated  with  the  maximum  eigenvalue  of  the 
resulting  matrix.  If  the  assumption  of  additivity  is  valid,  the  weights 


where: 

1. 

2. 

3. 

and  4. 
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should  be  reasonable  approximations  of  the  intensity  of  preference  of  the 
decision  maker.  Furthermore,  the  analytic  hierarchy  process  does  not  rely 
on  the  assumption  of  consistency  demanded  by  the  formal  utility  approach. 
Saaty  calculates  the  degree  of  inconsistency  and  makes  this  known  to  the 
decision  maker  who  can  then  alter  the  qualitative  comparison  to  make  the 
result  more  consistent. 

While  multiattribute  utility  analyses  can  incorporate  large  numbers  of 
value  attributes,  it  is  nevertheless  true  that  important  considerations  may 
be  omitted  by  such  analyses.  Implicit  in  this  argument  is  the  belief  that 
because  human  value  judgments  are  typically  based  on  large  numbers  of 
complex  considerations,  such  omissions  are  likely  to  be  common  rather  than 
rare.  A  large  body  of  psychological  research  suggests,  however,  that  the 
premise  of  this  argument--that  people  commonly  consider  many  factors  in 
forming  value  judgments--i s  false.  In  the  early  and  mid  1950 ' s ,  Miller 
(1956)  conducted  a  series  of  studies  relating  to  basic  human  memory  and 
judgment  processes.  He  concluded  that  people  are  capable  of  actively 
coding  and  manipulating  at  most  eight  to  ten  (and  usually  fewer)  "chunks" 
of  conceptual  information  at  any  given  time.  Siovic  and  Lichtenstein 
(1971)  concluded  that  in  virtually  every  case,  simple  linear  statistical 
models,  incorporating  three  or  fewer  independent  variables,  explained  over 
80  percent  of  the  system  variance  in  the  predictions  or  evaluations  of 
individual  experts.  Shepard  (1964)  suggested  that  people  accurately  recall 
that  they  considered  a  large  number  of  factors  during  the  course  of 
deliberating  over  an  important  issue,  but  fail  to  realize  that  they 
actively  evaluated  only  a  small  number  of  factors  at  the  real  moment  of 
deci  si  on. 
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Anderson  (1974)  and  Slovic  and  Lichtenstein  (1971),  after  reviewing 
the  extensive  literature  on  human  prediction,  evaluation,  and  choice, 
conclude  that  simple  additive  or  multilinear  models  typically  explain 
virtually  all  of  the  systematic  variance  in  human  judgment  processes. 

While  examples  of  more  complex  interactions  between  input  dimensions  have 
been  observed,  such  interactions  never  account  for  more  than  a  small 
fraction  of  the  systematic  variance  in  human  judgment  processes.  Thus,  the 
existing  literature  suggests  that  simple  additive,  multiplicative,  or 
multilinear  models  will  almost  always  provide  an  excellent  approximation  of 
intuitive  human  judgment  processes,  including  intuitive  assessments  of  the 
value  of  decision  outcomes. 

Anderson  and  others  have  used  the  functional  measurement  approach  to 
study  a  wide  variety  of  human  judgment  processes  ranging  from  purely 
perceptual  tasks  (estimating  the  weights  of  pairs  of  objects)  to  evaluating 
mul tiattri bute  choice  outcomes.  Their  findings  have  shown  that  over  a 
wide  range  of  contexts,  human  judgments  can  be  very  accurately  represented 
by  simple  algebraic  models.  This  literature  suggests  that  simple  algebraic 
MAU  models  of  the  type  considered  here  have  provided  excellent 
approximations  to  human  preferences  in  virtually  every  context  studied  by 
psychologists. 

The  Air  Force  has  attempted  in  many  studies  to  develop  manpower 
planning  models;  however  a  search  of  the  Air  Force  Manual  35-5  (1980) 
reveals  no  models  currently  accepted  by  the  Air  Force  for  such  planning. 

All  previous  attempts  did  not  have  adequate  statistical  validity  to  be 
placed  into  general  or  special  purpose  situations. 
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Once  a  satisfactory  method  is  developed  for  measuring  productivity, 
initial  psychometric  tests  must  be  conducted.  A  secondary  purpose  of  this 
paper  is  to  initially  assess  the  validity  of  the  productivity  measure. 
Simply  stated,  validity  is  the  extent  to  which  any  instrument  measures  what 
it  is  intended  to  measure.  While  there  are  several  different  types  of 
validity  measures,  construct  validation  was  chosen  due  to  the  lack  of  both 
criterion  measures  and  the  "acceptance  of  the  universe  of  content" 

(Cronbach  &  Meehl ,  1955,  p.  282)  in  measurement  of  productivity.  Indeed, 
productivity  measurement  is  particularly  elusive  in  many  "white  collar" 
work  centers  (Lockwood  &  Ludner,  1982).  However,  it  could  be  argued  that 
the  method  for  deriving  the  measure  provides  a  degree  of  content  validity. 
It  should  be  noted  that  criterion  validity  could  be  assessed  in  the  future 
over  an  extended  data  set  (period  of  time). 

Construct  validity  is  "concerned  with  the  extent  to  which  a  particular 
measure  relates  to  other  measures  consistent  with  theoretically  derived 
hypotheses  concerning  the  concepts  that  are  being  measured"  (Carmines  and 
Zeller,  1979,  p.  23).  Classically,  one  correlates  a  particular  measure 
with  a  second  measure  which  is  linked  theoretically  to  the  same  construct. 
If  there  is  a  strong  positive  correlation,  a  degree  of  support  is  provided. 

Carmines  and  Zeller  and  others  (Balch,  1974;  Sullivan,  1974)  suggest 
that  the  most  critical  requirement  in  determining  construct  validity  is  in 
establishing  the  theoretical  relationship  between  the  measures  of  the 
concepts.  It  follows  that  a  strong  relationship  to  more  than  one  indicator 
enhances  the  construct  validity. 
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Procedures 


Developing  the  Factors 

The  procedures  used  to  develop  a  productivity  model  followed  the 
following  general  process: 

1.  select  factors  which  drive  the  workload 

2.  weigh  those  factors  as  to  relative  importance  as  workload 
drivers 

3.  combine  the  weighted  factors  into  a  model 

4.  collect  data  to  establish  the  overall  model  and 

5.  validate  the  model. 

The  senior  executive  officer  determined  that  the  chiefs  of  System 
Programs  Offices  (SPO's)  under  him  would  have  the  greatest  impact  on  the 
development  of  the  productivity  measure.  During  the  development  of  such  a 
model  as  this,  it  is  imperative  that  the  top  managers  be  committed  to  the 
components  of,  and  use  of  such  a  model.  It  was  also  directed  that  only 
objective  attributes  be  considered  so  the  SPO's  could  not  game  the  system 
as  would  be  possible  if  subjective  attributes  were  used.  The  only 
potential  subjective  consideration  was  predetermined  to  be  the  issue  of 
complexity  which  would  be  applied  at  the  highest  decision  making  level. 
However,  for  the  purpose  of  this  validation,  no  subjective  allowance  for 
complexity  was  allowed. 

The  senior  manager  further  directed  a  modified  nominal  group  technique 
be  used  to  insure  that  all  SPO's  participate,  and  that  no  participant  would 
be  criticized  or  treated  negatively  for  his  or  her  contribution.  The 
process  involves  only  minor  time  saving  changes  from  the  typical  nominal 
group  technique.  The  factors  were  quickly  and  fairly  determined  with 
representation  from  all  key  management. 


Weighting  of  Factors 


A  weighting  process  was  chosen  because  it  provides  a  weight  the 
decision  maker  will  accept  and  assures  consistency.  The  decision  maker 
compared  on  paired  comparisons  on  a  1  to  9  scale.  Psychological  research 
suggests  that  a  7  or  9  scale  is  the  most  favorable  for  making  such 
comparisons.  Other  research  suggests  that  paired  comparisons  are  superior 
to  ranking  3  or  more  at  a  time.  Hence,  if  A  is  strongly  preferred  to  B  an 
entry  in  a  matrix  of  7  for  A  vs  B,  and  1/7  for  B  vs  A.  See  the  Appendix 
for  an  illustration  of  the  process  (Saaty,  1980). 

After  the  matrix  is  formed,  a  computer  program  computes  the  maximum 
eigenvalue  and  its  associated  normalized  eigenvector.  This  eigenvector 
represents  the  best  weighting  for  the  factors. 

The  consistency  factor  is  computed  by  the  formula: 

Cl  =  Consistency  Index  =  max  -  n 

n-1 

RI  =  Random  Index  was  determined  for  n  by  experiment  and  reported  in 
Saaty  (p.  31) 

Consistency  Ratio  =  CI/RI 

If  the  consistency  is  greater  than  1.00,  then  the  decision  maker  has 
probably  stated  preference  of  the  form  acb<cr  a.  Usually  this  situation 
will  be  quickly  resolved.  The  procedure  was  programmed  on  the  Tektronix 
4054  with  dynamic  graphics.  Dynamic  graphics  permits  a  grid  with  one 
blinking  line  to  indicate  which  comparison  is  furthest  off.  If  the 
consistency  is  between  .1  and  1.0,  the  inconsistency  may  be  only  one  of 
degree.  That  is  if  "a"  should  be  strongly  preferred  to  "b",  and  "b"  is 
strongly  to  "c",  then  "a"  should  be  very  strongly  preferred  to  "c".  Often 
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the  decision  maker  will  say  "a"  is  only  strongly  favored  to  "c".  After  the 
decision  maker  is  satisfied  with  his  or  her  weightings,  a  hard  copy  is 
made. 

In  this  case,  the  weights  for  the  participants  within  a  SPO  were 
averaged  to  get  a  composite  weight  for  the  SPO.  Further,  the  four 
composite  weights  of  the  SPO 1 s  were  averaged  to  arrive  at  an  overall  weight 
for  the  entire  acquisition  agency.  The  SPO 1 s  are  assumed  to  be 
sufficiently  homogenous  to  permit  averaging. 

Mathematical  Model 

The  model  for  this  study  will  value  the  following  form, 
n 

P  =  H.  Ki  Ui  (Xi) 
i=l 

where  P  is  the  productivity  value 

Ki  is  the  weight  of  attribute  Xi 
Xi  is  the  value  of  attribute  i 
U - (X^ )  is  the  marginal  ability  of  Xi 

The  ability  function,  Ui(Xi),  is  assumed  to  be  linear;  therefore 
Ui (Xi )  =  AiXi  where 

Ai  is  a  constant  which  associates  the  value  1  for  Ui(Xi)  where  Xi 
is  at  its  maximum  possible  value. 

The  factors  were  assumed  to  be  related  to  productivity  in  a  linear 
way.  Further  research  might  reveal  otherwise  for  some,  but  the  overall 
decision  is  not  likely  to  change.  The  highest  likely  value  of  each  item  of 
raw  data  was  assigned  the  value  one,  and  the  lowest  raw  value,  which  was 
zero  in  all  cases,  was  assigned  the  value  zero.  The  actual  percent  of  this 
high  value  for  each  factor  was  used  to  compute  a  productivity  measure. 
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The  second  assumption  was  that  the  weighted  factor  values  were 
combined  in  an  additive  way.  Research  suggests  that  most  of  the  variance 
in  a  decision  can  be  explained  by  the  additive  model.  This  additive  score 
reflects  the  overall  productivity  of  the  SPO  or  entire  agency.  No  check  of 
additive  assumption  was  made  due  to  the  limited  availability  of  the 
subjects . 

Complexity  is  a  subjective  factor  which  may  be  applied  by  a 
knowledgeable  decision  maker  before  SPO  comparisons  are  made.  However, 
during  the  current  validity  assessment,  no  allowance  was  made  for 
complexity  to  remove  all  subjectivity  during  testing. 

Raw  data  is  divided  by  the  number  of  management  personnel  assigned  to 
a  System  Program  Office.  This  produces  a  raw  value  per  person  (RPP).  In 
order  to  transform  the  data  as  described  above,  the  RPP  for  each  data  input 
is  divided  by  the  highest  likely  value  (or  max)  found  in  each  series  of 
data.  This  yields  a  percent  of  the  highest  value  which  is  then  multiplied 
by  the  weight  to  produce  a  productivity  measure  (PM). 

Organizational  Development  Toward  Enhanced  Productivity 

This  issue  of  productivity  measurement  has  had  hightened  attention  at 
this  organization  for  a  three-year  period.  During  that  time,  the 
organization  had  contracted  with  the  Leadership  and  Management  Development 
Center  (the  center  for  Air  Force  management  consultation)  for  a  continuing 
organizational  development  intervention.  The  immediate  challenge  was  to 
determine  and  refine  appropriate  indicators  of  productivity.  The 
consultation  effort  centered  around  a  survey  guided  development  program 
that  provided  direct  feedback  to  all  levels  of  supervision.  A  pictorial 
representation  of  the  program  is  depicted  in  Figure  1  and  briefly  described 
below.  Strategies  included  structured  data  feedback  at  the  individual 
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supervisor  level,  3-day  seminars  in  the  use  of  data  for  work  group 
enhancement  for  all  supervisors,  required  management  action  planning 
meetings  throughout  the  organization  over  a  6-month  period,  and  5-day 
seminars  targeted  for  project  management. 

The  survey  instrument  that  was  the  catalyst  that  the  supervisors  used 
to  produce  consistent  change  over  time  was  the  Organizational  Assessment 
Package  (OAP).  This  instrument  has  been  in  use  by  the  Air  Force  since 
1979.  The  various  psychometric  properties  of  the  instrument  have  been  well 
documented.  It  was  designed  jointly  by  the  Air  Force's  research  center 
(Human  Resources  Laboratory)  and  the  consultation  staff  (Leadership  and 
Management  Development  Center).  Its  development  and  initial  factor 
analysis  are  found  in  Hendrix  and  Halverson  (1979a;  1979b).  Several 
investigations  have  addressed  the  instruments  reliability  and  validity. 
Short  and  Hamilton  (1981)  found  good  test-retest  reliability  and  strong 
internal  consistency.  Further,  Short  and  Wilkerson  (1981)  provided  a 
measure  of  construct  validity.  Finally,  consistency  of  factor  structure 
was  demonstrated  across  functional  work  areas  (Hightower  and  Short,  1982), 
and  across  time  periods  (Hightower,  1982).  While  other  studies  are  in 
progress,  all  conducted  assessments  of  the  instrument  have  found  the  survey 
to  have  sound  psychometric  properties. 

During  the  development  of  the  previously  described  productivity 
measure,  LMDC  proceeded  with  a  planned  series  of  data  gatherings  as  shown 
in  Figure  1.  The  key  measures  from  the  OAP  are  further  described  in 
Appendix  A.  It  is  these  measures  that  were  independently  obtained  prior  to 
computation  of  the  "hard"  productivity  measure  that  were  correlated  using 
data  for  the  overall  organization  across  time.  As  suggested  in  the 
literature  (Franklin  &  Drexler,  1977),  the  organizational  behavior  data 
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YEAR  MEASURED  PRODUCTIVITY  DATA  POINTS. 


(Management-Supervision  and  Perception  of  Productivity)  were  correlated  in 
a  lead  fashion  with  the  outcome  measures  (measured  productivity). 

Specifically,  as  shown  in  Figure  1,  Time  1  data  were  linked  with 
fiscal  year  1981-1  (1st  half);  Time  2  data  linked  with  fiscal  year  1981-2 
(2nd  half);  Time  3  data  linked  fiscal  year  1982-2. 


/ 
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RESULTS 

Development  Results 

In  Step  1  of  the  procedure,  attributes  were  determined  in  a  meeting  of 
the  SPO  chiefs  and  the  senior  executive  officers.  Fourteen  factors  were 
identified  as  driving  the  workload  of  the  acquisition  agency.  These 
factors  are  organized  into  old  or  existing  contracts,  new  work  in  the 
current  year,  and  out  year  work  (Figure  2).  Appendix  B  further  develops 
the  attributes  by  defining  how  the  six  month  value  of  each  should  be 
computed.  All  SPO 1 s  used  these  standard  definitions. 

In  Step  2,  participants  in  the  nominal  group  technique  followed  the 
weighting  procedure  to  establish  the  weights  shown  in  Table  1. 

It  should  be  noted  that  the  last  several  attributes  could  be 
eliminated  without  changing  the  final  decision.  However,  it  was  decided  to 
keep  all  the  attributes  for  psychological  reasons,  as  well  as  potential 
future  changes  in  model  area  emphasis.  Additionally,  some  would  object  if 
they  thought  their  manpower  was  based  on  only  four  or  five  factors. 

Validity  Findings 

Based  on  the  weighted  factors,  productivity  measures  were  computed  for 
the  highest  aggregated  management  level.  These  figures  are  shown  in 
Table  2. 

As  described  earlier,  survey  data  were  linked  with  measured  productivity 
with  an  approximate  6  month  lead.  Measures  of  the  corresponding  lead 
Organization  Assessment  survey  data  are  shown  in  Table  3.  Therefore,  since 
attitudinal  measures  were  not  available,  measured  productivity  for  the 
first  hair  of  fiscal  year  1982  are  not  used  in  this  analysis.  However,  it 
should  be  noted  that  the  data  appear  to  fit  with  a  general  notion  that  pro¬ 
ductivity  was  constantly  enhanced  during  this  two  year  period  (Figure  3). 
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HIERARCHY  OF  FACTORS 


FIGURE  2 


1.  Number  of  Contracts 

2.  RDT&E  Expenditures  Per  Month 

3.  Production  Expenditures  Per 
Month. 

4.  Dollar  Value  of  the  Unliquid 
Obligations  or  undelivered 
balances  on  RDT&E  Contracts 

5.  Value  of  Undelivered  Balances 
on  the  Production  Contracts 


6.  Number  of  Source  Selections  in 
Progress 

7.  Number  of  RFPs  Issued 

8.  Number  of  Contracts  Awarded 

9.  Total  RDT&E  Target  Prices  of 
Contracts  Awarded 

10.  Total  Target  Production  Prices 
of  Contracts  Awarded 


11.  Number  of  RDT&E  Projects 


Elements  Managed 

12.  Number  of  Production  Projects 

FUTURE 

13.  RDT&E  Dollars  In  FYDP 

14.  Production  Dollars  In  The  FYDP 
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Table  1 


Productivity  Values 

Using  1  Oct  -  31  Mar  FY81  Actuals 
(Limited  Sample) 


FACTOR  WGT 


1 

0.249 

2 

0.125 

3 

0.052 

4 

0.070 

5 

0.058 

6 

0.136 

7 

0.040 

8 

0.082 

9 

0.056 

10 

0.052 

11 

0.020 

12 

0.012 

13 

0.027 

14 

0.023 

MAX 

RAW 

2.0 

98.0 

0.2 

6.5 

0.1 

3.6 

0.6 

54.2 

1.9 

106.4 

0.2 

3.0 

0.7 

15.0 

0.9 

19.0 

1.4 

50.5 

4.3 

116.3 

0.3 

25.0 

0.3 

25.0 

11.0 

555.1 

100.0 

3164.3 

RPP  PM 


0.88 

0.100 

0.06 

0.036 

0.03 

0.017 

0.48 

0.056 

0.95 

0.029 

0.03 

0.018 

0.13 

0.008 

0.17 

0.015 

0.45 

0.018 

1.04 

0.013 

0.22 

0.015 

0.22 

0.009 

4.96 

0.012 

28.25 

0.006 

TOTAL 


0.326 


NOTE:  Dollars  in  Millions 

112  Personnel  Assigned 
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Table  2 


Measured  Productivity  at  the  Highest  Aggregated 
Management  Level  by  Half  Fiscal  Years 


Year 

1981-1 

1981- 2 

1982- 1 

1982-2 

Measured  Productivity 

.392 

.444 

.497 

.488 

Table  3 

Organizational  Assessment  Survey  Data  at  the 

Highest  Management  Level 

Time 

Perceived  Productivity 

Management-Supervision 

Sept  1980 

5.33 

4.76 

April  1981 

5.62 

5.31 

March  1982 

5.78 

5.49 
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Figures  3  and  4  show  the  general  relationship  of  measured  productivity 
with  perceived  productivity  and  management  effectiveness.  Pearson  Product 
Moment  correlations  are  displayed  in  Table  4.  It  should  be  noted  that  con¬ 
current  perceived  productivity  (as  measured  at  the  end  of  the  6  month  data 
periods  as  opposed  to  6  months  lead)  also  correlates  extremely  well 
(r  =  .947). 
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TIME  PERIODS 


FIGURE  3.  MEASURED  PRODUCTIVITY  ACROSS  TIME  (INCLUDING  THE  FIRST  HALF  OF 
FISCAL  YEAR  1982) 


MANAGEMENT/  PERCEIVED 
SUPERVISION  PRODUCTIVITY 


TIME  PERIODS 


FIGURE  4.  MANAGEMENT/SUPERVISION  AND  PERCEIVED  PRODUCTIVITY 
ACROSS  TIME. 


i 
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Table  4 


Correlations  Between  Measured  Productivity 

and 

the  Perception  of  Productivity  and  Management-Supervision 


Index 

Correlation  Coefficient 

Perceived  Productivity  (lead) 

.993* 

Management-Supervision  .972* 

Perceived  Productivity  (concurrent)  .947* 


*p  .01 
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Discussion 


It  was  the  intention  of  the  authors  to  present  a  strategy  for 
measuring  productivity  at  a  major  development  and  acquisition  organization, 
to  provide  an  initial  assessment  of  the  validity  of  the  measure,  and 
finally,  to  present  a  methodology  for  determining  productivity  indicators 
in  other  areas  that  do  not  lend  themselves  to  simplistic  productivity 
measurement. 

The  initial  validity  assessment  is  extremely  strong.  According  to 
Cronbach  (1970)  it  is  unusual  for  criterion  validity  coefficients  to  rise 
above  .60.  While  we  have  used  a  construct  validation,  it  is  still  accurate 
that  any  positive  correlation  indicates  that  predictions  from  the 
measurement  are  more  accurate  than  guesses.  While  additional  forthcoming 
data  points  will  provide  a  stronger  check  of  the  validity,  these  initial 
data  are  encouraging. 

This  type  measurement  should  have  direct  applicability  to  other 
development  and  acquisition  organizations.  More  importantly,  however,  is 
the  possibility  of  using  the  modified  Analytic  Hierarchy  Process  as  a  tool 
for  determining  productivity  measurements  in  other  "white  collar"  work 
centers.  By  acquiring  the  experience  and  input  of  the  workers  and 
management  in  combination  with  the  logically  designed  Analytic  Hierarchy 
Process,  few  systems  will  defy  quantification,  and  thus  control. 

A  potential  problem  of  the  described  measurement  indicator  is  the  lack 
of  direct  measurement  of  effectiveness.  Cook  (1982)  clearly  summarizes  the 
literature  as  requiring  some  balance  in  the  equation  between  efficiency  and 
effectiveness.  Interestingly,  the  attitudinal  measures  (Perceived 
Productivity  and  Management-Supervision)  both  have  direct  links  to 
effectiveness  (i.e.,11  ...  quality  of  output  of  my  workgroup";  etc.) 
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Therefore,  it  is  possible  that  the  derived  measure  of  productivity  does 
indirectly  capture  a  degree  of  effectiveness.  Nevertheless,  most  models 
should  have  some  indicator  of  effectiveness. 

Further  assessment  of  the  validity  of  this  particular  measure  is  the 
logical  extension  of  this  paper.  First,  similar  work  can  be  conducted  at 
the  next  level  below  this  highest  aggregate  to  begin  assessing  the  validity 
by  major  program  office.  This  will  provide  more  understanding  of  the 
nature  of  the  indicators,  as  well  as  the  applicability  at  lower  working 
1 evel s . 

Second,  additional  data  points  will  soon  become  available.  The 
validity  data  will  be  enhanced  by  further  data  points  (this  is  not  to  say 
that  the  correlations  will  become  more  positive,  rather  that  more 
credibility  can  be  placed  on  a  larger  data  set). 

It  should  be  noted  that  further  practical  assessment  should  be 
conducted  at  this  agency.  Since  the  process  has  been  in  use  in  excess  of 
one  year,  designs  could  be  developed  to  determine  overall  effectiveness, 
reliability  and  other  forms  of  validity. 

Finally,  extended  use  of  the  AHP  in  other  organizations  merits 
investigation  as  many  managers  are  faced  with  no  sound  alternative.  As 
more  experience  is  gained  with  these  procedures,  modifications  can  lead  to 
a  strengthening  of  the  process.  The  cost  benefit  of  a  valid  productivity 
measurement  should  be  obvious.  What  remains  is  the  future  scrutiny  of  the 
new  approach  that  may  provide  the  key  to  productivity  measurement  in  those 
"white  collar"  hard  to  measure  work  centers. 


23/24 


References 


Anderson,  N.  H.,  "Algebraic  Models  in  Perception,"  Handbook  of 

Perception,  Vol .  2,  edited  by  E.  C.  Cattergate  and  M.  P.  Friedn 
New  York,  Academic  Press,  1974. 

Anderson,  N.  H. ,  "Information  Integration  Theory:  A  Brief  Survey," 

Developments  in  Mathematical  Psychology,  Vol.  2,  Edited  by  D.  H 
Krantz,  R.  C.  Atkinson,  D.  Luce  and  P.  Suppes,  San  Francisco, 
California,  Freeman  Press,  1974. 

Balch,  G.  I.,  "Multiple  Indicators  in  Survey  Research:  The  Concept 

Sense  of  Political  Efficiency."  Political  Methodoloty,  1974,  1 
1-44. 

Bell,  D.  E.,  "Consistent  Assessment  Procedures  Using  Conditional 

Utility  Functions,"  Operations  Research,  Vol.  27,  No.  5  (1979). 

Bodily,  S.  E. ,  The  Utilization  of  Frozen  Red  Cells  in  Blood  Banking 
Systems:  A  Decision  Theoretic  Approach,  Technical  Report  No. 
94,  Operations  Research  Center,  Massachusetts  Institute  of 
Technology,  Cambridge,  Mass.  (1974). 

Carmines,  E.  G.  and  Zeller,  R.  A.  Reliability  and  Validity.  London: 
Sage  Publications,  1979. 

Cook,  J.  W.  Productivity:  Its  Improvement  and  Measurement  (A _ 

Literature  Review]")  (Air  Command  and  Staff  College  Student 
Report,  82-0575),  Maxwell  AFB,  AL:  Air  University,  1982. 

Cronbach,  L.  J.  Essentials  of  Psychological  Testing,  3rd  Edition. 

New  York:  Harper  and  Row,  1970.  ’  ~ 

Cronbach,  L.  J.  and  Meehl ,  P.  E.  Construct  Validity  in  Psychological 
Tests.  Psychological  Bulletin,  1955,  52,  281-302. 

deNeufville,  R.  and  R.  L.  Kenney,  "Use  of  Decision  Analysis  in  Airport 
Development  for  Mexico  City,"  in  Analysis  of  Public  Systems,  A 
W.  Drake,  et  al . ,  editors,  M.I.T.  Press,  Cambridge,  Mass.,  1972 

Fishburn,  P.  E.,  "Independence  in  Utility  Theory  with  Whole  Product 
Sets,"  Operations  Research,  Vol.  13,  1965. 

Fox,  William,  "Modified  Nominal  Group  Technique"  Unpublished  paper 
University  of  Florida,  Gainesville,  FL. 

Franklin,  J.  L.  &  Drexler,  J.  A.,  Jr.  Organizational  Conditions  as 

Predictors  of  unit  level  effectiveness.  Paper  presented  at  the 
American  Psychological  Association  85th  annual  meeting,  San 
Francisco,  August,  1977. 


25 


Hendrix,  W.  H.  &  Halverson,  V.  B.  Situational  factor  identification  in 
Air  Force  organizations  (AFHRL-TR-79-10) .  Brooks  AFB ,  TX:  ATr 
Force  Human  Resources  Laboratory,  1979(a). 

Hendrix,  W.  H.  &  Halverson,  V.  B.  Organ i zational  survey  assessment 
package  for  Air  Force  organizations  (AFJR: -TR-78-93) .  Brooks 
AFB,  TX:  Air  Force  Human  Resources  Laboratory,  1979(b). 

Hightower,  J.  M.  Temporal  stability  of  the  factor  structure  of  the 
Organizational  Assessment  Package  (LMDC-TR-82-2).  Maxwell  AFB, 
AL:  Leadership  and  Management  Development  Center,  1982. 

Hightower,  J.  M.  &  Short,  L.  0.  Factor  stability  of  the 

Organizational  Assessment  Package  (LMDC-TR-82-1 ) .  Maxwell  AFB, 
AL:  Leadership  and  Management  Development  Center,  1982. 

Huber,  G.  P. ,  "Multiattribute  Utility  Models:  A  Review  of  Field  and 
Field-Like  Studies,"  Management  Science,  Vol .  20  (1974), 
pp.  1393-1402. 

Keeney,  R.  L.  &  H.  Raiffa,  Decisions  with  Multiple  Objectives: 
Preferences  and  Value  Tradeoffs,  Wiley  and  Sons,  Inc.,  New 
York,  New  York:  1976. 

Lockwood,  M.  &  Ladmer,  A.  The  Performance  Model:  An  Objective 

Approach  to  Measuring  White  Collar  Productivity.  Manage,  1982, 
October,  30-32. 

Miller,  G.  A.,  "The  Magical  Number  Seven,  Plus  or  Minus  Two:  Some 
Limits  on  our  Capacity  to  Process  Information,"  Psychological 
Review,  Vol.  63  (1956). 

Saaty,  Thomas  L.  &  Vargas,  Luis  G.  The  Logic  of  Priorities,  Klower  - 
Nijhoff  Publishing  Co.  1982. 

Saaty,  Thomas  L.,  Decision  Making  for  Leaders,  Wadsworth  Inc.,  1982. 

Shepard,  R.  N.,  "On  Subjectively  Optimum  Selection  Among  Multi¬ 
attribute  Alternatives,"  Human  Judgment  and  Optimality. 

Edited  by  M.  W.  Shelley  &  G.  L.  Bryan,  New  York:  Wiley,  1964. 

Short,  L.  0.  &  Hamilton,  K.  L.  An  examination  of  the  reliability  of 
the  Organizational  Assessment  Package  (LMDC-TR-81-2) .  Maxwell 
AFB,  AL:  Leadership  and  Management  Development  Center,  1981. 

Short,  L.  0.  &  Wilkerson,  D.  A.  An  examination  of  the  construct 

validity  of  the  Organizational  Assessment  Package.  Proceedings 
of  the  23rd  Annual  Conference  of  the  Military  Testing 
Association"!  (Vol  II,  pp.  1089-1098).  Arlington,  VA:  U.S.  Army 
Research  Institute,  1981. 


26 


Slovic,  P.  &  S.  Lichtenstein.  "Comparison  of  Bayesian  and  Regression 
Approaches  to  the  Study  of  Information  Processing  in  Judgment." 
Organizational  Behavior  and  Human  Performance,  Vol .  6  (19711. 
pp.  649-744. 

Sullivan,  J.  L.  Multiple  Indicators:  Some  Criteria  of  Selection.  In 
H.  M.  Blalock  (ed)  Measurement  in  the  Social  Sciences: 

Theories  and  Strategies.  Chicago:  AVC,  1974. 

Thorn,  M.  E.  Perspectives  on  the  Issue  of  Productivity.  Armed  Forces 
Comptroller,  1980,  25_,  30-34. 


27/28 


APPENDIX  A 


_ Selected  Organizational  Assessment  Package  Scales 

Management  Supervision: 


STATEMENT 

NUMBER 

58 

59 

60 
61 
62 

63 

64 

65 


STATEMENT 

My  supervisor  is  a  good  planner. 

My  supervisor  sets  high  performance  standards. 

My  supervisor  encourages  teamwork. 

My  supervisor  represents  the  group  at  all  times. 

My  supervisor  establishes  good  work  procedures. 

My  supervisor  has  made  his  responsibilities  clear  to  the 
group. 

My  supervisor  fully  explains  procedures  to  each  group 
member. 

My  supervisor  performs  well  under  pressure. 


Perception  of  Productivity 

STATEMENT 

NUMBER  STATEMENT 

The  quanti ty  of  output  of  your  work  group  is  very  high. 

78  The  qua! ity  of  output  of  your  work  group  is  very  high. 

l\hen  high  priority  work  arises,  such  as  short  suspenses, 
crash  programs,  and  schedule  changes,  the  people  in  my 
work  group  do  an  outstanding  job  in  handling  these 
situations. 

80  Your  work  group  always  gets  maximum  output  from  available 
resources  ( e . g . ,  personnel  and  material). 

81  Your  work  group's  performance  in  comparison  to  similar 
work  groups  is  very  high. 
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APPENDIX  B 


ATTRIBUTE  DEFINITIONS 


FACTOR  NO.  ITEM  TITLE  DEFINITION 


1  Number  of  Contracts  The  number  of  contracts  (C)  are 

equal  to  total  active  contracts  managed  at  the  beginning  of  the 
fiscal  year  of  (Cf)  plus  contracts  awarded  (Ca)  during  the  fiscal 
year  less  contracts  completed  (Cc)  during  the  fiscal  year.  A 
contract  is  completed  when  all  items  and  data  are  delivered. 

c  =  (Cf  +  Ca)  -  Cc 

2  RDT&E  Monthly  Expenditures  RDT&E  monthly  expenditures 

(Me)  are  equal  to  total  RDT&E  expenditures  (TRe)  for  the  year, 
plus  the  total  of  other  RDT&E  expenditures  (TOe)  for  the  year, 
divided  by  12. 

Me  =  TRe  +  TOe 
12 

3  Production  Monthly  Expenditures  Production  monthly  expendi¬ 
tures  (Me)  are  equal  to  total  Production  expenditures  (TPe)  for 
the  year  pi  us  the  total  of  other  Production  expenditures  (TOe) 
for  the  year  divided  by  12. 

Me  =  TPe  +  TOe 
12 

4/5  ULOs  (RDT&E  and  Production)  The  unliquidated  obligations 

for  RDT&E  and  Production  funds  are  the  difference  of  Obligations 
(Ob)  and  Expenditures  (Eb)  at  the  beginning  of  the  year  plus  the 
difference  of  the  anticipated  Obligations  (Oe)  and  Expenditures 
(Ee)  for  the  end  of  the  year  divided  by  2. 

(Ob  -  Eb)  +  (Oe  -  Ee) 

2 

(ITEMS  6  -  10  DEAL  ONLY  WITH  THE  PRESENT  FY) 

6  Number  of  Source  Selections  In  Process  The  number  of  Source 

Selection  in  process  are  equal  to  the  total  number  of  source 
selections  in  process  during  the  fiscal  year.  A  source  selection 
begins  when  AFSC  Program  Direction  requiring  source  selection  is 
issued.  A  source  selection  is  completed  when  a  contract  is 
awarded.  Down  selection  which  requires  SSEB  and  SSEC  procedures 
are  source  selections. 
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12 


13 


14 


NO.  ITEM  TITLE  DEFINITION 


Noncompetitive  RFPs  issued  Total  Noncompetitive  RFPs 

issued  are  equal  to  the  Total  number  of  RFPs  issued  during  the 
fi seal  year.  Restructuring  an  active  contract  may  be  included  if 
a  RFP  is  issued  with  new  SOW,  specifications,  CDRLs,  and  delivery 
schedule.  A  MIPR  sent  is  not  a  noncompetitive  RFP  issued. 

Number  of  Contracts  Awarded  The  total  number  of 

contracts  awarded  are  equal  to  the  number  of  contracts  awarded 
during  the  fiscal  year.  A  MIPR  sent  is  a  contract  awarded  if  the 
MIPR  includes  SOW,  specifications,  a  CDRL,  delivery  schedule, 

CSSR  or  CPR  reporting  on  cost  based  contracts,  and  terms  and 
conditions;  i.e.,  if  a  MIPR  sent  has  all  the  elements  of  a 
contract  the  MIPR  can  be  included  as  a  contract  awarded. 

Total  RDT&E  target  prices  of 

Contracts  Awarded  Total  RDT&E  target  prices  of 

contracts  awarded  are  the  sum  of  the  target  prices  of  all  RDT&E 
contracts  awarded  including  options  during  the  fiscal  year. 

Total  Production  target  prices  of 

Contracts  Awarded  Total  target  prices  of 

production  contracts  awarded  are  the  sum  of  the  target  prices  of 
all  production  contracts  awarded  including  options  during  the 
fiscal  year. 

Number  of  RDT&E  Projects  Managed  Number  of  RDT&E  projects 
managed  are  equal  to  the  total  number  of  RDT&E  programs, 
projects,  and  tasks  requiring  separate  budget  documents, 
reporting,  and  separate  contracts.  A  RDT&E  project  implemented 
by  a  MIPR  sent  should  be  included. 

Number  of  Production  Projects 
Managed  Number  of  production 

projects  are  equal  to  the  total  number  of  production  programs  and 
projects  requiring  separate  budget  documents,  reporting,  and 
separate  contracts.  A  production  project  implemented  by  a  MIPR 
sent  should  be  included. 

RDT&E  dollars  in  the  FYDP  RDT&E  dollars  in  the  FYDP 

are  equal  to  the  total  RDT&E  dollars  in  the  President's  defense 
budget.  The  FYDP  covers  a  five  year  period  starting  with  the 
next  FY  plus  four.  (In  the  case  of  FY  82  the  period  was  equal  to 
FY  83  through  FY  87). 

Production  dollars  in  the  FYDP  Production  dollars  in  the 

FYDP  are  equal  to  the  total  production  dollars  in  the  President's 
defense  budget.  The  FYDP  covers  a  five  year  period  starting  with 
the  next  FY  plus  four.  (In  the  Case  of  FY  82  the  period  was 
equal  to  FY  83  through  FY  87). 
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